Counter-balancing vibrations from a vehicle for stabilizing image capture

ABSTRACT

Embodiments are described for a stabilization system configured, in some embodiments, for stabilizing image capture from an aerial vehicle (e.g., a UAV). According to some embodiments, the stabilization systems employs both active and passive stabilization means. A passive stabilization assembly includes a counter-balanced suspension system that includes an elongated arm that extends into and is coupled to the body of a vehicle. The counter-balanced suspension system passively stabilizes a mounted device such as an image capture device to counter motion of the UAV while in use. In some embodiment the counter-balanced suspension system passively stabilizes a mounted image capture assembly that includes active stabilization means (e.g., a motorized gimbal and/or electronic image stabilization). In some embodiments, the active and passive stabilization means operate together to effectively stabilize a mounted image capture device to counter a wide range of motion characteristics.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/579,624, entitled “COUNTER-BALANCED SUSPENDED IMAGE STABILIZATIONSYSTEM,” filed Sep. 23, 2019; which is a continuation application ofU.S. patent application Ser. No. 15/790,776, entitled “COUNTER-BALANCEDSUSPENDED IMAGE STABILIZATION SYSTEM,” filed Oct. 23, 2017, and issuedas U.S. Pat. No. 10,455,155 on Oct. 22, 2019; which is entitled to thebenefit of and/or the right of priority to U.S. Provisional PatentApplication No. 62/412,770, entitled “COUNTER-BALANCED SUSPENDED IMAGESTABILIZATION SYSTEM,” filed Oct. 25, 2016, each of which is herebyincorporated by reference in its entirety for all purposes. Thisapplication is therefore entitled to a priority date of Oct. 25, 2016.

TECHNICAL FIELD

The present disclosure generally relates passive and active imagestabilization systems. Specifically, the present disclosure relates tosystem configured to stabilize image capture from an aerial vehicle suchas UAV across a wide range of motion characteristics.

BACKGROUND

Unmanned Aerial Vehicles (UAVs) generally include any aircraft capableof controlled flight without a human pilot onboard. UAVs may becontrolled autonomously by onboard computer processors and/or by aremotely located human pilot. Like pilot-driven helicopters, some UAVscan be configured as rotor-based aircraft. For example, severalmanufacturers offer commercially available UAVs that include fourrotors, otherwise known as “quadcopters.” Often UAVs are fitted withimage capture devices such as cameras that can be configured both tocapture images (including video) of the surrounding environment andincreasingly to facilitate autonomous visual navigation. Often themotion of a UAV in flight can negatively impact the quality of imagecapture. Accordingly, systems can be employed to counter such motionthrough active and passive means.

BRIEF DESCRIPTION OF THE DRAWINGS

The present embodiments are illustrated by way of example and are notintended to be limited by the figures of the accompanying drawings. Inthe drawings:

FIG. 1A shows an example UAV that includes an image capture devicesuspended below the body of the UAV;

FIG. 1B shows an example UAV that includes an image capture devicemounted to the front side of the body of the UAV via a cantilever mount;

FIG. 2 shows a top view of an example UAV that incorporates one or moreof the novel stabilization techniques described herein;

FIG. 3 shows a cross section of the example UAV depicted in FIG. 2 ;

FIGS. 4A-4C show a series of cross sections of the example UAV depictedin FIG. 2 that further illustrate passive stabilization of an imagecapture assembly by a counter-balanced suspension system;

FIG. 5 shows a detail of the cross section of the UAV depicted in FIG. 2that illustrates components of an example image capture assembly;

FIG. 6 is a diagram that illustrates how various types of stabilizationsystems can be employed to counter motion across a range of frequencies;

FIG. 7 shows a pair of example bode plots that illustrate how rotationalmotion can result from translation motion at a range of frequencies in agiven kinematic system;

FIG. 8A shows an isometric view of example UAV similar to the UAVdepicted in FIG. 2 that incorporates one or more of the novelstabilization techniques described herein;

FIG. 8B shows a top view of the example UAV depicted in FIG. 8A;

FIG. 8C shows a top view of the example UAV depicted in FIG. 8A with thehousing hidden to reveal the arrangement of components related to imagestabilization systems;

FIG. 8D shows a front view of the example UAV depicted in FIG. 8A;

FIG. 9A shows a side view of a passive stabilization assembly configuredfor use with the UAV depicted in FIG. 8A;

FIG. 9B shows a top view of the passive stabilization assembly depictedin FIG. 9A;

FIG. 9C shows a front isometric view of the passive stabilizationassembly depicted in FIG. 9A;

FIG. 9D shows a rear isometric view of the passive stabilizationassembly depicted in FIG. 9A;

FIG. 9E shows a side view of the passive stabilization assembly depictedin FIG. 9A in the context of the housing of the UAV depicted in FIG. 8A;

FIG. 9F shows a top view of the passive stabilization assembly depictedin FIG. 9A in the context of the housing of the UAV depicted in FIG. 8A;

FIG. 9G shows an isometric view of the passive stabilization assemblydepicted in FIG. 9A in the context of the housing of the UAV depicted inFIG. 8A;

FIG. 10A shows an isometric view of an isolator configured for use withthe passive stabilization assembly depicted in FIG. 8A;

FIG. 10B shows a side view of the isolator depicted in FIG. 10A;

FIG. 10C shows a cross-section view of the isolator depicted in FIG.10A;

FIG. 11A shows a side view of an image capture assembly coupled to thepassive stabilization assembly depicted in FIG. 9A;

FIG. 11B shows a top view of the image capture assembly of FIG. 11Acoupled to the passive stabilization assembly depicted in FIG. 9A;

FIG. 11C shows a first front isometric view of the image captureassembly of FIG. 11A coupled to the passive stabilization assemblydepicted in FIG. 9A;

FIG. 11D shows a second front isometric view of the image captureassembly of FIG. 11A coupled to the passive stabilization assemblydepicted in FIG. 9A;

FIG. 11E shows a first rear isometric view of the image capture assemblyof FIG. 11A coupled to the passive stabilization assembly depicted inFIG. 9A;

FIG. 11F shows a second rear isometric view of the image captureassembly of FIG. 11A coupled to the passive stabilization assemblydepicted in FIG. 9A;

FIG. 12A shows a side view of the image capture assembly depicted inFIG. 11A with a housing cover;

FIG. 12B shows a top view of the image capture assembly depicted in FIG.11A with a housing cover;

FIG. 12C shows an isometric view of the image capture assembly depictedin FIG. 11A with a housing cover;

FIG. 12D shows a side view of the image capture assembly depicted inFIG. 11A, with a housing cover, and in the context of the housing of theUAV depicted in FIG. 8A;

FIG. 12E shows a top view of the image capture assembly depicted in FIG.11A, with a housing cover, and in the context of the housing of the UAVdepicted in FIG. 8A;

FIG. 13 shows a detail of the image capture assembly of FIG. 11A;

FIG. 14 is a flow diagram illustrating an example process for activeimage stabilization, according to some embodiments;

FIG. 15 shows a conceptual diagram of a localization and navigationsystem for guiding navigation and image/video capture by a UAV;

FIG. 16 shows a conceptual diagram of a system for estimating theposition and/or orientation of a UAV using a network of phased arraywireless transceivers;

FIG. 17 shows a conceptual diagram of an example system for passivelocalization of an object tracked by a UAV;

FIG. 18A-18B illustrate example methods for estimating the positionand/or orientation of objects using computer vision technology; and

FIG. 19 shows a high level system diagram of components in an exampleUAV.

DETAILED DESCRIPTION

Specific embodiments of the invention are described herein for purposesof illustration. Various modifications may be made without deviatingfrom the scope of the invention. Accordingly, the invention is notlimited except as by the appended claims.

Overview

Aerial vehicles, such as UAVs can be fitted with image capture devices(e.g., one or more cameras) to capture images (including video) of asurrounding physical environment while the vehicle is in flight. Variousimage stabilization techniques can be implemented in an attempt tocounter the motion of a vehicle while in flight in an attempt to improvethe quality of image capture. For example, many currently availableimage capture devices include sensors (e.g., accelerometers and/orgyroscopes) configured to detect motion such as changes in positionand/or orientation. Using this motion information, a number oftechniques may be employed to actively stabilize image capture tocounter the detected motion. For example, in some cases image capturedevices may include integrated mechanical systems configured to actuatecertain optical elements (e.g., optical sensors and/or the lens) tocounter the detected motion of the image capture device. In the case ofdigital image capture devices, software may alternatively oradditionally be employed to transform the captured digital images tocounter the motion of the image capture device. Such techniques aregenerally referred to as electronic image stabilization (EIS).

While image stabilization systems internal to the image capture devicecan counter relatively small changes in position/orientation they havelimited effectiveness countering more drastic changes inposition/orientation, for example those experienced by a vehicle inflight. To counter such motion, a system can be employed to stabilizethe body of the image capture device relative to the body of thevehicle. This can be achieved, for example by mounting the image capturedevice body to a mechanical gimbal system configured to rotate the imagecapture device about one or more axes relative to the body of an aerialvehicle.

FIG. 1A shows a first example configuration of a UAV 100 a that includesan image capture device 102 a suspended below the body of the UAV 100 a.In this example configuration, the image capture device 102 may bemounted to a bottom side of the body of UAV 100 a via a multi-axismechanical gimbal 103 a configured to rotate the image capture device102 a about multiple axes to counter a motion of the UAV 100 a while inflight. To counter higher frequency translations (e.g., vibrationscaused by the propulsion systems onboard the UAV), such a configurationmay also include passive motion isolators 104 a between the body of theUAV 100 a and the mechanical gimbal system 103 a supporting the imagecapture device 102 a. While the system described in FIG. 1A can beconfigured to effective stabilize image capture to some degree itsignificantly impacts the overall form factor of the vehicle (as isevident in the depiction in FIG. 1A). This may not be as a primaryconcern for aircraft that are much larger than the image capture device(e.g., a manned aircraft), but does become more of a concern forrelatively small vehicles such as a quadcopter UAV. Further, in such aconfiguration, the field of view of the image capture device 102 a maybecome obscured at certain angles by elements of the body of the UAV 100a, for example the landing supports 106 a shown in FIG. 1A.

To address the form factor issue, an image capture device can instead bemounted in a cantilevered configuration relative to the body of thevehicle. For example, FIG. 1B shows an image capture device 102 bcoupled to the front side of the body of UAV 100 b via a cantilevermount. Although not depicted in FIG. 1B, such a configuration may alsoinclude stabilization systems such as the mechanical gimbal 103 a andvibration isolators 104 a shown in FIG. 1A. However, a cantilevermounted image capture device 102 b does introduce challenges withrespect to passive motion isolation. Any vibration isolators placedbetween the body of the UAV 100 b and the cantilever mounted imagecapture device 102 b should be stiff enough to handle the shear forcecaused by the weight of the image capture device 102 b, but soft enoughto dampen translational motion in the body of the UAV 100 b along arange of frequencies. Vibrational isolators (e.g., shock absorbingmounts made of rubber or an elastomer material) alone will have limitedeffectiveness isolating the image capture device 102 b against certainmotion because the material characteristics need to isolate such motionwill cause the image capture device 102 b to sag under its own weight.

Introduced herein are novel techniques for stabilizing image capturefrom an aerial vehicle that address the issues discussed above. Forexample, embodiments are described that include a counter-balancedsuspension assembly configured to passively isolate an image capturedevice from certain motion of the body of an aerial vehicle in flight.Specifically, according to some embodiments a counter-balancedsuspension assembly may include an elongated arm that extends into aninterior space of the body of the aerial vehicle and is dynamicallymounted to the body via one or more isolators. The elongated arm ineffect acts as a counter balance to the weight of the image capturedevice resulting in a dynamically balanced suspension system for theimage capture device that has minimal impact on the overall factor ofthe vehicle. Further, in some embodiments, the counter-balancedsuspension assembly can be combined with one or more activestabilization techniques (e.g., mechanical gimbals and/or EIS) tofurther improve image stabilization capability to counter a range ofmotion profiles.

Note that embodiments are described herein in the context of a UAV,specifically a UAV configured as quadcopter, to provide clearillustrative examples, however the described techniques are not limitedto such applications. A person having ordinary skill will appreciatethat the described techniques can be similarly applied to any platformsin motion. For example, similar image stabilization systems asintroduced herein may be applied to other types of manned and unmannedaerial vehicles (e.g., fixed-wing jet aircraft, fixed-wing propelleraircraft, rotorcraft, airship, etc.), land vehicles (automobiles,motorcycles, bicycles, rail vehicles, etc.), and water vehicles (ships,boats, hovercraft, etc.). Further, embodiments are described herein inthe context of stabilizing a mounted image capture device or imagecapture assembly, however the described techniques are not limited tosuch applications. The techniques for passive and active stabilizationdescribed herein can in many cases be easily applied to stabilizing anyother type of device or object. For example, the described techniquesmay be implemented to stabilize a mounted payload container, sensordevice, communications system, weapons system, illumination system,propulsion system, industrial tool (e.g., a robotic arm), etc.

FIG. 2 shows a top view of an example UAV 200 that may include one ormore of the image stabilization techniques described herein. As shown inFIG. 2 , example UAV 200 includes a body housing 210 and a stabilizedobject 240 extending from one side (e.g., the front side of UAV 200).For clarity the stabilized object 240 will be described herein in thecontext of an “image capture assembly,” however as previously mentioned,this stabilized object can be any type of object. Also shown in FIG. 2are rotor assemblies 280 mounted on opposing sides of the body housing210. Each rotor assembly may include one or more rotors and in somecases a perimeter structure substantially extending around the rotorblades. A perimeter structure can protect the one or more rotors fromcontact with objects in the physical environment, while UAV 200 is inflight and in some embodiments may house sensors (e.g., optical sensors)used for autonomous navigation. The concept of a perimeter structure isdescribed in more detail in U.S. application Ser. No. 15/164,679,entitled, “PERIMETER STRUCTURE FOR UNMANNED AERIAL VEHICLE,” filed May25, 2016, the contents of which are hereby incorporated by reference intheir entirety. Note, rotor assemblies 280 are illustrated in FIG. 2 toprovide structural context for example UAV 200, but as indicated bytheir rendering in dotted line are otherwise not essential to the imagestabilization techniques described herein.

The body housing 210 of example UAV 200 is shown in FIG. 2 asrectangular when viewed from above suggesting a cuboid structure,however it shall be understood that housing 210 may have any shape andbe of any dimension. In general, housing 210 may include walls thatenclose an interior body space (not shown in FIG. 2 ). For example, thearea of housing 210 that is viewable in FIG. 2 may be a top wall.

FIG. 3 shows a cross section of the example UAV 200 depicted in FIG. 2 .The location of the view in FIG. 3 is indicated in FIG. 2 by view arrowsmarked with the number “3.” A detail of the cross section shown in FIG.3 is depicted in FIG. 5 as indicated by the dotted line box 290. Asmentioned with respect to FIG. 2 and as shown in more detail in thecross section of FIG. 3 , housing 210 may include one or more wallssurrounding an interior space 218 of the housing 210. The interior space218 has an opening 220 at the “front end” of the housing 210 throughwhich the image capture assembly 240 protrudes and is defined by theinterior surfaces of one or more of the walls of the housing 210. Forexample, as shown in FIG. 3 , the interior space 218 is defined by aninterior top surface 214, an interior bottom surface 216 opposite theinterior top surface 214, and an interior back surface 212 opposite theopening 220 and located towards the “back end” 222 of the housing 210.

As further shown in FIG. 3 , image capture assembly 240 is structurallycoupled to a passive stabilization assembly that includes an elongatedarm 232, a mounting assembly 236, and one or more isolators 234. Animage capture assembly 240 may include one or more components related toimage capture systems including, but not limited to an image capturedevice (e.g., a camera) and one or more active stabilization systems(e.g., mechanical gimbals and/or EIS systems) configured to activelystabilize image capture by the image capture device. In the depictedembodiment, the elongated arm 232 has a proximal end and a distal endand is arranged within the interior space 218 to extend from theinterior back surface 212 towards the opening 220 at the front end ofthe housing 210. The distal end of the elongated arm is dynamicallycoupled to the interior back surface 212 via one or more isolators 234and the proximal end is coupled to the mounting 234 assembly which is inturn mounted to one or more of the interior top surface 214 or interiorbottom surface 216 (not depicted in FIG. 3 ) via one or more isolators234. The image capture assembly 240 is structurally coupled to themounting assembly 236 of the passive stabilization assembly.Accordingly, the passive stabilization assembly and image captureassembly form a structural unit dynamically coupled to the housing 210.Further, as is evident in FIG. 3 and as will be described in more detailwith respect to FIGS. 4A-4C, elongated arm 232 forms a counter balanceto the mass of the components in the image capture assembly 240. Theimage capture assembly 240 is thereby stabilized by a counter-balancedsuspension system.

Note that the arrangement of elements comprising example UAV 200 aredepicted in FIG. 3 in a simplified form to clearly illustrate theconcept of passive stabilization of a mounted image capture assembly 240through the use of a counter-balanced suspension system. For example,housing 210 is depicted in a simplified rectangular form, but dependingon the specific implementation, housing 210 may have any shape of anydimension. Further, the walls forming the housing 210 are depicted in asimplified form and are not to be construed as limiting with regard toarrangement or dimension. For example, interior back surface 212 isdepicted in FIG. 3 as being part of an interstitial wall arranged atappoint between the front and back end of the housing 210. A personhaving ordinary skill will recognize that this wall need not be presentin all embodiments. For example, in some embodiments the elongated arm232 may simply extend to a surface of a wall at the back end 222 of thehousing 210. In other embodiments, the interior back surface 212 may bepart of a support structure other than a wall within the housing 210(e.g., a beam, a plate, a mounting bracket, etc.).

Further, the elements of the passive stabilization assembly are depictedin a simplified and illustrative purpose, and should not be construed aslimiting with respect to arrangement or dimensions. For example,elongated arm 232 is depicted as uniform in dimension and extending alittle over half way along the length of the housing 210. However, thisis only an example embodiment. The actual implementation in any vehiclewill depend greatly on the geometry of the vehicle housing 210, thecharacteristics of the image stabilization assembly 240, and theparticular image stabilization requirements. As another example,mounting assembly 236 is depicted as a discrete component coupling theimage capture assembly 240 to the elongate arm 232. However. in otherembodiments, the passive stabilization assembly may include fewer ormore components than as shown. For example, the elongated arm 232 maysimply extend from the image capture assembly 240. Also, the passivestabilization assembly is shown in FIG. 3 as dynamically coupled to thehousing 210 at two points (at least in the cross section view), howeverthis is not to be construed as limiting. The passive stabilizationassembly may be dynamically coupled to the housing at fewer or morepoints and at different locations than as shown while remaining withinthe scope of the currently described innovations. A person havingordinary skill will recognize that the coupling points will depend onthe geometry of the housing 210 and the various components of thepassive stabilization assembly and image capture assembly 240 for anygiven implementation.

FIGS. 4A-4C show a series of cross sections similar to the cross sectiondepicted in FIG. 3 that further illustrate passive stabilization of animage capture assembly 240 by a counter-balanced suspension system. FIG.4A, for example, shows a cross section of example UAV 200 in a restingstate with the dynamic components (i.e., the passive stabilizationassembly and image stabilization assembly 240) in mechanicalequilibrium. For example, the dynamic components supported or suspendedvia isolators 234 may have a center of mass at point 260. Note that thelocation of the center of mass 260 in FIG. 4A is an example provided forillustrative purposes and is not to be construed as limiting. Forexample, the center of mass 260 need not be located at or about themounting assembly 236 or near an isolator 234 as shown in FIG. 4A. Thespecific arrangement in any given embodiment will depend on thestabilization requirements, geometry of the system, the materials used,etc.

FIGS. 4B-4C show the passive stabilization of the image assembly 240 inresponse movement of example UAV 200. For example, FIGS. 4B-4Cillustrate stabilization of image assembly 240 in response to therotational and/or translational motion by example UAV 200. The examplesshown in FIGS. 4B-4C are provided for illustrative purposes and do notnecessarily show actual ranges of motion for the components of exampleUAV 200. For example, the depicted changes in position/orientation maybe exaggerated for clarity.

As shown, the isolators 234 may in some embodiments act as springdampers to isolate the dynamic components from certain rotational and/ortranslational motion by UAV 200. For example, in some embodiments eachisolator 234 may act as a spring damper to isolate motion in all of thex, y, and z directions. As will be explained, in some embodiments eachisolator 234 may exhibit, based on its geometry and material properties,a 1:1:1 ratio of compression stiffness to tensile stiffness to shearstiffness. In other words, each isolator 234 may act as a spring damperthat responds uniformly in the x, y, and z directions.

Generally speaking, an increase in the length of the elongated arm 232will tend to increase the moment of inertia of the dynamic componentsabout a center of rotation (for example, but not necessarily the centerof mass 260). This increase in the moment of inertia will tend to resistexternal torque applied through the motion of the housing, therebyproviding a stabilizing effect. Accordingly, in some embodiments,elongated arm 232 extends all the way to or at least as close aspossible to the back side 222 of housing 210. In some embodiments, thelength of the elongated arm 232 may be limited due to space constraints.For example, the cross sections shown in FIGS. 3 and 4A-4C showelongated arm 232 extending approximately ⅔ of the length of housing 210with the remaining ⅓ reserved for housing other functional componentsincluding, but not limited to batteries, computer processing systems,etc.

As previously mentioned, FIG. 5 shows a detail of the cross sectionshown in FIG. 3 as indicated by the dotted line box 290. Specifically,FIG. 5 shows, in simplified form, an example image capture assembly 240that includes a motorized gimbal mechanism for actively stabilizing amounted image capture device (e.g., a camera) 248. In general, amotorized gimbal mechanism may include multiple link arms coupled at oneor more motorized rotation joints. The link arms and rotation jointsform a mechanical linkage coupling the image capture device 248 to thepassive stabilization assembly (e.g., at mounting assembly 236). Inresponse to the detected motion (e.g., using accelerometers or othermotion sensors) motors at the rotation joints actuate the link armsabout the axes of rotation of the rotation joints to counter thedetected motion(s). The combined effect of this actuation by the motorsis to stabilize the mounted image capture device 248 relative to aparticular frame of reference (e.g., the surface of the Earth). FIG. 5shows an example embodiment of a motorized gimbal mechanism that isrotatable about two axes using two gimbal motors 242 and 244. As shownin FIG. 5 , image capture device 248 can be pitched up and down byactuating motor 244 and rotated by actuating motor 242. In someembodiments, such a two-axis gimbal system may be implemented as part ofa hybrid mechanical-digital gimbal system which is described in moredetail with respect to FIGS. 11A-11F.

This two-axis configuration is described for illustrative purposes, butis not to be construed as limiting. In some embodiments image captureassembly 240 may include a motorized gimbal providing more or fewerdegrees of freedom of motion for mounted image capture device 248.

FIG. 6 is a diagram that illustrates how various types of stabilizationsystems can be employed to counter motion across a range of frequenciesof motion. For example, as shown in the diagram of FIG. 6 , activestabilization techniques may generally be more effective at relativelylower frequencies whereas passive stabilization techniques may generallybe more effective at relatively higher frequencies. In the exampleembodiment of quadcopter UAV (e.g., similar to example UA 200) activestabilization techniques (e.g., mechanical stabilization of the imagecapture device using a gimbal and/or EIS) may effectively stabilizeimage capture at or below frequencies in the area of 15 Hz. Note thatthis is just an example observation and would not necessarily apply toall embodiments of the presently described innovations. The range ofeffectiveness of any given stabilization system will depend on a numberof implementation-specific design factors.

Returning to the example of a quadcopter UAV, active stabilizationsystems may be less effective at stabilizing motion above approximately15 Hz for a number of reasons. For example, in any active stabilizationsystem (mechanical or EIS) some degree of latency is likely introducedbased on processing of received motion sensor data, generating responsecommands, and either processing images (EIS) or actuating gimbal motors.This latency may reduce the overall effectiveness of countering motionat higher frequencies (e.g., high frequency vibration introduced by therotors of a quadcopter UAV in operation). Further, higher frequencymotion will generally be associated with lower translationaldisplacement (e.g., high frequency vibration). Active mechanicalstabilization of a mounted image capture device may be less effective atcountering such small translational motions due to the limitedpositional accuracy of the motors used in such systems. For example,typical stepper motors that may be utilized in a motorized gimbalmechanism are accurate to about ±0.10°. EIS can also run into problemswhen attempting to counter high frequency motion due to the nature inwhich the image is captured at optical sensor. In many digital imagecapture systems (e.g., CMOS) an image is captured at the optical sensorby rapidly scanning across a given field of view (either vertically orhorizontally). Due to the time required to scan across the field ofview, rapid motion in the scene (e.g., due to high frequency vibration)can lead to a “wavy” effect in the captured images. This effect can insome cases be alleviated with further image processing, however there isa processing efficiency benefit to passively isolating the image capturedevice from such motion before image capture.

Passive image stabilization, on the other had can be more effective athandling higher frequency motion such as vibration. For example, in thecase of a quadcopter UAV similar to UAV 200, the aforementionedcounter-balanced suspension system may be effective at isolating amounted image capture device from translational motion at frequenciesbeyond the effective range (e.g., above 15 Hz) of an integrated activesystem. It will be appreciated that due to its unique geometry, theaforementioned counter-balanced suspension system will exhibit a widereffective range that, for example simply mounting the image captureassembly to the UAV housing using vibration isolators.

As also noted in FIG. 6 , at very low frequencies, vehicle controls maybe utilized to a degree to further stabilize image capture. For example,as will be described in more detail, in some embodiments a UAV may beconfigured for autonomous navigation utilizing one or more localizationand flight planning systems. Such system may be configured to prioritizethe stability of the airframe platform when performing maneuvers with agoal of providing stable image capture. Similarly, in the case of amanned craft or remotely-controlled craft using “fly-by-wire” systems,pilot control inputs can be interpreted and corresponding controlcommands generated to maneuver the craft in a stable manner to enablequality image capture.

Accordingly, to counter a wide range of motion characteristics (e.g.,translational motion across a across a wide range of frequencies), animage stabilization system may be implemented that employs both passiveand active stabilization techniques, for example as described withrespect to FIGS. 3-5 . Such systems may be generally referred to as ahybrid active-passive stabilization systems.

FIG. 7 shows a pair of example bode plots that illustrate how rotationalmotion can result from translation motion at a range of frequencies in agiven kinematic system. A kinematic system with a given set ofcharacteristics (e.g., geometry, materials, etc.) will have a frequencyor set of frequencies at which the system will tend to oscillate in theabsence of a driving or damping force. This is generally referred to asthe “natural frequency” and as shown in FIG. 6 , can lead to extremespikes in oscillating motion at certain frequencies. The plots providedin FIG. 6 , are examples provided to illustrate this concept but do notnecessarily pertain to any of the systems or components describedherein. A person having ordinary skill will understand that inimplementing an embodiment of the present innovation, certaincharacteristics (e.g., geometry, materials) may be adjusted to reducethe effects of such aforementioned spikes across a range of frequenciesof translational motion.

FIGS. 8A-13 show a series of views that illustrate in greater detail aparticular embodiment of an example UAV 800 (e.g., similar to UAV 200described with respect to FIGS. 2-5 ) that incorporate some of theaforementioned image stabilization techniques.

FIG. 8A is an isometric view of example UAV 800 in the form of aquadcopter. Similar to UAV 200 described with respect to FIGS. 2-5 , UAV800 includes a central body housing 810 with a forward facing imagecapture assembly 840 that includes an image capture device for capturingimages (including video) of the surrounding physical environment whileUAV 800 is in flight. As shown in FIG. 8A, in this example embodiment,UAV 800 includes rotor assemblies on opposing sides of the central bodyhousing 810. Each rotor assembly includes one or more rotors 882 thatare protected by a perimeter structure 880, substantially extendingaround the blades of the rotor assembly 880. Perimeter structure 880 canprotect the one or more rotors 882 from contact with objects in thephysical environment, while UAV 800 is in flight and in some embodimentsmay house sensors 884 (e.g., optical sensors) used for autonomousnavigation. FIG. 8B shows a top view of example UAV 800 that furtherillustrates how the perimeter structure 880 extends around the blades ofthe rotors 882. The concept of a perimeter structure is described inmore detail in U.S. application Ser. No. 15/164,679, entitled,“PERIMETER STRUCTURE FOR UNMANNED AERIAL VEHICLE,” filed May 25, 2016,the contents of which are hereby incorporated by reference in theirentirety. Note, the rotors are illustrated in FIG. 8A to providestructural context for example UAV 800, but are otherwise not essentialto the image stabilization techniques described herein.

Similarly described with respect to UAV 200, the housing 810 of UAV 800may include one or more walls surrounding an interior space of thehousing 810. The interior space has an opening at the “front end” of thehousing 810 through which the image capture assembly 840 protrudes andis defined by the interior surfaces of one or more of the walls of thehousing. The walls of the housing 810 and perimeter structure 880 can bemade of one or more structural components made of any material orcombination of materials that have strength and weight characteristicssuitable for use in an aircraft. For example, the walls of housing 810and perimeter structure 880 can be made of plastic, metal (e.g.,aluminum), carbon fiber, synthetic fiber (e.g., Kevlar®), or some sortof composite material such as carbon or glass fiber embedded in an epoxyresin. Specifically, in example UAV 800, the walls of housing 810 andperimeter structure 880 may be made of a plurality of plastic structuralcomponents formed through an injection molding and/or 3-D printingprocess. The plurality of components can be assembled and fastened toeach other using any of integrated clips, screws, bolts, glue, welding,soldering, etc.

FIG. 8C shows a top view of example UAV 800 similar to the top viewshown in FIG. 8B, except that the walls of the housing 810 are hidden toshow the relative arrangement of components related to the imagestabilization systems. As shown in FIG. 7C, UAV 800 includes a passivestabilization assembly 830 arranged within the interior space of housing810. Coupled to the passive stabilization assembly 830 is the imagecapture assembly 840 which can include various active stabilizationcomponents that are described in more detail later. Forward mountedimage capture assembly and associated passive stabilization assembly 830arranged within the interior space of housing 810 allows for the lowprofile of UAV 800 as evident in the front view of the vehicle shown inFIG. 8D. Contrast the profile of UAV 800 shown in FIG. 8D with theprofile of UAV 100 a shown in FIG. 1A.

FIGS. 9A-9G show a series of detailed views of the example passivestabilization assembly 830 shown in FIG. 8C. Specifically, FIG. 9A showsa side view, FIG. 9B shows a top view, FIG. 9C shows a front isometricview, and FIG. 9D shows a rear isometric view. Similarly, FIGS. 9E, 9F,and 9G show a side view, top view, and isometric view (respectively) ofexample passive stabilization assembly 830 in the context of the housing810 of UAV 800.

As show in FIGS. 9A-9D, similar to the passive stabilization assembly ofUAV 200 described with respect to FIGS. 2-4C, passive stabilizationassembly 830 includes an elongated arm 832, a mounting assembly 836 anda plurality of isolators 834 through which the assembly 830 is coupledto interior surfaces of the housing 810.

In some embodiments, elongated arm 832 is a cylindrical structure of acertain length, for example as shown in FIG. 9A. Note however, thatexample elongated arm 832 is shown in FIGS. 9A-9D as being a straightcylinder-shaped member with generally uniform thickness. This example isprovided for illustrative purposes, but should not be construed aslimiting. In other embodiments the “elongated arm” may not be a singlemember or may have a different shape. For example, to accommodategeometry constraints within the interior space of housing 810, theelongated arm 832 has a proximal end and a distal end. The proximal endis coupled to the mounting assembly 836 and the distal end isdynamically coupled to an interior surface (e.g., an interior backsurface 812, shown in FIGS. 9E-9F) of housing 810 via an isolator 834.As shown in FIGS. 9A-9D, the distal end of elongated arm 832 may becoupled to the isolator 834 via a mounting clip 837. Example elongatedarm 832 can be made of any material or combination of materials thathave strength and weight characteristics suitable for use in a UAV suchas UAV 800. For example, elongated arm can be made of plastic, metal(e.g., aluminum), carbon fiber, synthetic fiber (e.g., Kevlar®), or somesort of composite material such as carbon or glass fiber embedded in anepoxy resin and may be formed using any process appropriate for theselected material including injection molding, 3-D printing, machining,etc.

As with mounting assembly 236 described with respect to UAV 200,mounting assembly 836 is configured to dynamically couple to an interiorsurface (e.g., an interior top surface 814, shown in FIG. 9E) of housing810 via isolators 834. Specifically, example, mounting assembly 836includes a first mounting assembly arm 836 a extending laterally from anaxis of the elongated arm 832 and a second mounting assembly arm 836 bextending laterally from the axis of the elongated arm 832, opposite thefirst mounting assembly arm 836 a, for example as shown in the top viewprovided in FIG. 9B. At each of the arms 836 a and 836 b, the mountingassembly 836 is dynamically coupled to an interior surface (e.g., aninterior top surface) of housing 810 via an isolators 834. Accordingly,example passive stabilization assembly 830 is dynamically coupled to thehousing 810 of UAV 800 at three points. This provides a balancedconfiguration, but is not necessary in all embodiments. In otherembodiments, the passive stabilization assembly may be dynamicallycoupled to housing 810 at fewer or more points and at differentlocations. For example, each arm 836 a-b may include two isolators 834,one coupled a top interior surface and one coupled to a bottom interiorsurface. Alternatively, in some embodiments, the mounting assembly 836may include more than the two arms shown in FIGS. 9A-9D.

As with isolators 234, isolators 834 may in some embodiments act asspring dampers to isolate the dynamic components (i.e., passivestabilization assembly 830 and the mounted image capture assembly 840)from certain rotational and/or translational motion by UAV 800. Forexample, in some embodiments each isolator 834 may act as a springdamper to isolate motion in each of the x, y, and z directions.Isolators 834 are described in more detail with respect to FIGS.10A-10C, however generally speaking isolators 834 may be formed of anelastomer material and based on their geometry and the properties of theelastomer material may exhibit a 1:1:1 ratio of compression stiffness totensile stiffness to shear stiffness. In other words, each isolator 834may act as a spring damper that responds uniformly in the x, y, and zdirections. Note, in FIGS. 9A-9D, each isolator 834 has a uniformconfiguration. This may help with manufacturing and part replacementefficiency, but is not necessary in all embodiments. For example, insome embodiments the isolator coupling the elongated arm 832 to housing810 may be of a first type and the isolators coupling the mountingassembly 836 to housing 810 may be of a different type. A person havingordinary skill will recognize that this is a design consideration andwill be affected by the geometries and arrangement of dynamic portionswith respect to housing 810.

As further shown in FIGS. 9A-9D, one or more of the isolators 834 may beassociated with a corresponding mechanical stopper 835 configured tolimit the range of motion of the passive stabilization assembly 830relative to the housing 810 of UAV 800. The mechanical stoppers 835 maybe included to prevent interference or contact in general between thecomponents of the passive stabilization assembly 830 (and any mountedimage capture assembly 840) with other components associated withexample UAV 800. As shown in FIGS. 9A-9D, in an embodiment themechanical stoppers 835 include pegs of some type (e.g., made out ofplastics, metal, etc.) that are arranged within an open interior spaceof the isolators 834. Note, the specific geometry of the exampleisolators 834 is more readily apparent in FIGS. 10A-10C. In otherembodiments, the mechanical stoppers 835 may be place at any other pointrelative to the passive image stabilization assembly 830 to effectivelylimit motion of the assembly. However, an added benefit to arranging thestopper 835 as shown in FIGS. 9A-9D is that when assembly 830 is inmotion and reaches the said limit, the stopper 835 contacts the interiorsurface of the elastomer isolator 834 resulting in a soft stop insteadof contacting a rigid surface (e.g., of elongated arm 832) which (if notpadded) may result in a loud sound and/or damage to the components.

It will be appreciated that the passive stabilization assembly 830depicted in FIGS. 9A-9D is an example provided for illustrative purposesand is not to be construed as limiting. In other embodiments, a passivestabilization assembly may include more or fewer discrete componentsthan as shown. For example, in an embodiment the elongate arm 232,mounting bracket 237, and mounting assembly 236 may collectivelycomprise a single part, for example formed through an injection molding,machining, or 3-D printing process.

FIGS. 10A-10B show a series of views of an example isolator 834, forexample as shown in FIGS. 9A-9D. Specifically, FIG. 10A shows anisometric view of isolator 834, FIG. 10B shows a side view of isolator834, and FIG. 10C shows a cross section (as indicated by cross sectionlabel 10C in FIG. 10A) of isolator 834. As shown in FIGS. 10A-10C,isolator 834 is generally cylindrical in nature but includes uniquegeometry configured to achieve the previously mentioned 1:1:1 ratio ofcompression stiffness to tensile stiffness to shear stiffness. Currentlyavailable passive vibration isolators typically include a solid portionof elastic material, for example in the form of a pad, that is placedbetween two rigid components. Conversely, as shown in FIGS. 10A-10Cexample isolator 834 is formed to include structural elements thataffect spring and damping characteristics. The unique geometry ofisolator 834 is more readily apparent when viewed in cross section inFIG. 10C. As shown in FIG. 10C, isolator 834 is cylindrical in natureand has a hollow portion extending along axis 1010. The walls formingthe structure of isolator 834 about axis 1010 can conceptually beseparated into a top portion 1012 configured to couple to a firstsurface (e.g., top interior surface 814 of housing 810) and a bottomportion 1014 configured to couple to a second surface (e.g., that ofmounting assembly 836). The two portions 1012 and 1014 are joined at acenter portion 1016 that can include one or more angled members that arearranged to act as spring dampers in each of the x, y, and z directions.Note that as shown in FIGS. 10A-10C, isolator 834 includes four suchangled members symmetrically distributed about axis 1010 with open spacebetween each member. This arrangement may be implemented to save onmaterial costs and/or to achieve desired spring/dampeningcharacteristics, however is not necessary in all embodiments. Forexample, depending on the material chosen, the center portion 1016 mayinclude a continuous wall about axis 1010 much like the top portion 1012and bottom portion 1014. Further the angles of the members of the centerportion 1016 are exemplary and will differ based on the particularrequirements of a given implementation.

As mentioned, in some embodiments isolators 834 may be made of one ormore elastomer materials (e.g., natural and/or synthetic rubbers). Ingeneral, the selected material should be suitable for forming intocomplex geometries (e.g., isolator 834 shown in FIGS. 10A-10C), andshould exhibit relatively low stiffness and relatively high dampingcharacteristics. With respect to the damping characteristics, in theexample embodiment of isolator 834, may exhibit a tangent delta (i.e.,energy loss factor) in the order of 0.6 and above, a beta in the orderof 3 and above, and/or a rebound elasticity of approximately 30% orless. As an example, use of Elastosil® R 752/50 as an elastomer materialalong with the geometry of isolator 834 described with respect to FIGS.10A-10C may provide suitable stiffness and damping characteristics topassively stabilize image capture assembly 840 with respect to housing810. Note that this material and the recited example stiffness anddamping characteristics may work for example isolators 834 in theexample embodiment of UAV 800, but do not necessarily apply to allembodiments of the presently described innovations. For example, theappropriate damping characteristics for a given isolator will heavilydepend on the characteristics of the object (e.g., image capture device)to be stabilized, the stabilization requirements of the givenimplementation, and the expected motion characteristics to be countered.A person having ordinary skill will recognize that these are designconsiderations that will change for each embodiment.

FIGS. 11A-11F show a series of detailed views of an image captureassembly 840 coupled to the example passive stabilization assembly 830shown, for example, in FIGS. 9A-9G. Specifically, FIG. 11A shows a sideview, FIG. 11B shows a top view, FIG. 11C shows a first front isometricview, FIG. 11D shows a second front isometric view, FIG. 11E shows afirst rear isometric view, and FIG. 11F shows a second rear isometricview.

As show in FIGS. 11A-11F, example image capture assembly 840 includes animage capture device (e.g., one or more digital cameras) and an activestabilization system in the form of a motorized gimbal mechanism (asindicated by gimbal motors 842 and 844). As previously discussed anactive stabilization system may be effectively utilized to counter lowerfrequency/higher magnitude changes in position/orientation of UAV 800.In this sense, the active stabilization systems of assembly 840 may worktogether with the passive stabilization provided by assembly 830. Note,however, that active stabilization may not be necessary in allembodiments. For example, it is contemplated that in some embodimentsimage capture assembly includes only an image capture device 848 coupledto passive stabilization assembly 830. In such embodiments, the imagecapture device 840 may or may not have its own internal activestabilization systems (e.g., EIS and mechanical stabilization of theoptical sensor/lens).

The motorized gimbal mechanism of assembly 840 shown in FIGS. 11A-11F issimilar to as described with respect to the detail of FIG. 5 . That isto say that the mechanism shown in example assembly 840 includes twomotors 842 and 844 that serve as rotation joints in a mechanical linkagecoupling the image capture device 848 to the passive stabilizationassembly 830. In response to detected motion (e.g., by any of housing810, assembly 830, or assembly 840), the motors 842, 844 are actuated(i.e., rotated) to adjust the orientation of image capture device 848about one or two axes to counter the motion. For example, as shown inFIGS. 11A-11F, motor 842 would rotate image capture device 848 about afirst axis extending along the length of housing 810 and motor 844 wouldrotate image capture device 848 about a second axis perpendicular to thefirst axis, thereby adjusting the pitch of image capture device 848relative to housing 810.

In some embodiments motors 842 and/or 844 may comprise a brushlesselectric motor. Brushless electric motors typically include a rotorcomponent with permanent magnets and a stator component that includescoiled conductors that form electromagnets. As electrical current isapplied through the coils of the stator component with the resultingelectromagnetic force interacting with the permanent magnets of therotor component, thereby causing the rotor component to rotate. In someembodiments, motors 842 and/or 844 may comprise a specific type ofbrushless motor commonly referred to as an “outrunner” motor. An“outrunner” motor can generally be understood as a type of brushlesselectric motor that spins an outer shell around its windings as opposedto just spinning a rotor axle. For example, an outrunner brushlesselectric motor may include a stator assembly coupled to a rotorassembly. The stator assembly may include a generally cylindrical statorhousing coupled to and surrounding a stator stack that includes multiplestator coils (e.g., made of copper) and optionally stator teeth that candivide an induced electromagnet into multiple sections. The stator stackmay be arranged about an axle bearing. Similarly, the rotor housing mayinclude a generally cylindrical housing coupled to and surrounding anaxle configured to be placed within the axle bearing of the statorhousing. The rotor housing further includes permanent magnets arrangedto be in close proximity with the stator stack when the motor isassembled. As current is applied through the coils of the stator stack,and electromagnetic fore is induced, which in turn causes the rotorassembly to rotate about the axle (due to the opposing magnetic forcecaused by the affixed permanent magnets). Brushless electric motorsprovide an accurate means for making fine adjustments to the positionand/or orientation of a mounted image capture device 848. However, aperson having ordinary skill will recognize that other types of motorsmay be implemented depending on the particular requirements of a givenembodiment.

In some embodiments, this two-axis motorized gimbal configuration may bepart of a hybrid mechanical-digital gimbal system that mechanicallyadjusts the orientation of the image capture device 848 about one or twoaxes while digitally transforming captured images (e.g., using EIS) tosimulate changes in orientation about additional axes. Further in someembodiments, a hybrid mechanical-digital gimbal system may beimplemented with fewer motors than as shown in FIGS. 11A-11F. Forexample, consider an image capture assembly similar to assembly 840, butthat includes only motor 844. In this example, motor 844 may handleactive adjustments in the pitch of image capture device 848 whileadjustments in roll and yaw are handled digitally, for example, byprocessing the captured digital images to rotate (roll) and pan (yaw)the field of view. Still further, in some embodiments, particularlythose in which UAV 800 includes autonomous navigation capabilities, theUAVs flight controls may be implemented as part of the hybrid mechanicaldigital gimbal system. For example, consider again an image captureassembly that includes only motor 844. Again, motor 844 might handleactive adjustments in the pitch of image capture device 848 while rolladjustments are handled digitally in the captured images. Yawadjustments, in this example, could then be handled by the flightcontrol systems through changing the orientation of the entire aerialplatform (i.e., housing 810 of UAV 800). Such a configuration may bebeneficial because it reduces the mechanical complexity of the system(only one gimbal motor), and reduces the image storage and processingrequirements (only rotational transforms and no panning).

An example hybrid mechanical-digital gimbal system has been describedfor illustrative purposes, but is not to be construed as limiting. Otherhybrid mechanical-digital gimbal systems may be arranged other than asdescribed above. For example, depending on the implementation, in someembodiments, it may be beneficial to handle pitch adjustments digitallyand roll adjustments mechanically. Further, a hybrid mechanical-digitalgimbal mechanism is not a necessary feature in all embodiments. Forexample, as previously mentioned, in some embodiments, the image capturedevice 848 may be simply coupled directly to the passive stabilizationassembly 830. In other embodiments, image capture device 848 may becoupled to the passive stabilization assembly 830 via a motorized gimbalwith more than two degrees of freedom (e.g., a three-axis or six-axisgimbal).

In some embodiments, image capture assembly 840 includes a housing thatsurrounds and protects the active components (e.g., motors 842, 844, andimage capture device 848). FIGS. 12A-12E show a series of views of theimage capture assembly 840 described with respect to FIGS. 11A-11F butincluding a protective housing 849. Specifically, FIG. 12A shows a sideview, FIG. 12B shows a top view, FIG. 12C shows an isometric view, FIG.12D shows a side view in the context of UAV housing 810, and FIG. 12Eshows a top view in the context of UAV housing 810. As with housing 810,housing 849 may comprise one or more structural components made of anymaterial or combination of materials that that have strength and weightcharacteristics suitable for use in an aircraft. For example, housing849 can be made of plastic, metal (e.g., aluminum), carbon fiber,synthetic fiber (e.g., Kevlar®), or some sort of composite material suchas carbon or glass fiber embedded in an epoxy resin. Specifically, inexample UAV 800, the housing 849 may be made of a plurality of plasticstructural components formed through an injection molding and/or 3-Dprinting process. The plurality of components can be assembled andfastened to each other using any of integrated clips, screws, bolts,glue, welding, soldering, etc.

In some embodiments system components associated with the operation ofUAV 800 may be mounted to dynamic portions of the vehicle (e.g., imagecapture assembly 840) to better balance the dynamic portions. Forexample, FIG. 13 shows a detail of the image capture assembly 840 (withhousing 849 removed) showing certain components (e.g., a computingboard) 841 mounted to the assembly. In general, such components arelikely to be mounted to “static” portions of UAV 800 (e.g., housing810). However, mounting such components 841 to a dynamic portion (e.g.,image capture assembly 840) may in some cases help to balance theoverall stabilization system. For example, recall that the elongated arm832 of passive stabilization assembly 830 servers as a counter weight toany mounted device or assembly (e.g., image capture assembly 840).Depending on the design constraints in a specific embodiment, it may bebeneficial to mount additional system components 841 to thecounterbalanced device or assembly (image capture assembly 840).Further, in some embodiments components 841 may not even be associatedwith the stabilization systems of assembly 840. For example, it iscontemplated that in some embodiments, components 841 may includecomponents (e.g., processing units, sensors, memory units, communicationdevices) associated with autonomous localization and navigation system(described in more detail later).

Example Active Image Stabilization Process

FIG. 14 is a flow diagram illustrating an example process 1400 foractive image stabilization that may be performed by systems associatedwith example image capture assembly 840 in some embodiments. Forillustrative clarity, certain process steps are described as beingperformed by components shown in FIG. 19 . This is provided forillustrative purposes and is not to be construed as limiting certainprocess steps to certain components. As an example, process 1400 beginsat step 1402 by detecting one or more motion sensors (e.g.,accelerometers 1926 and/or IMU 1928), motion associated with UAV 800with respect to one or more frames of reference. For example, motion maybe detected by motion sensors (e.g., accelerometers) mounted at one ormore of any of housing 810, passive image capture assembly 830, motors842, 844, and image capture device 848. Motion in any of these frames ofreference may further be estimated based on calculations performed by anautonomous localization and navigation system associated with UAV 810(described in more detail later).

In response to the detected motion, at step 1404 sensor data is outputby the one or more motion sensors and relative positions/motion arecalculated based on the sensor data. Note in some embodiments, the oneor more sensors may output at step 1404 raw sensor data that is thenprocessed by a separate processing component (e.g., processors 1912) tomake position/motion calculations. In some embodiments, the sensorsthemselves may process raw sensor data and output motion/positional datathat is based on the raw sensor data.

In response to calculating motions/positions, at step 1405, controlcommands/signals may be generated (based on the calculatedmotions/positions) that are configured to cause the one or more motors(e.g., motor(s) 842, 844) to actuate one or more rotation joints so asto stabilize a mounted image capture device (e.g., device 848) relativeto a particular frame of reference (e.g., the surface of the Earth). Insome embodiments, generation of control commands and/or signals may beperformed by one or more controller devices or other processing units(e.g., gimbal motor controllers 1907 and/or processors 1912). Forexample, in one embodiment, one or more processor(s) 1912 may generatecontrol commands based on the calculated motion/position that areconfigured to be read by a separate gimbal motor controller 1907. Thegimbal motor controller 1908 may interpret the control commands andbased on those control commands generate control signals that cause themotor(s) 842, 844 to actuate. For example, control signals in thiscontext may simply include applied voltage to induce electrical currentwithin the stator coils of a brushless motor.

As previously mentioned, in some embodiments active image stabilizationmay include electronic image stabilization (EIS). Accordingly, inresponse to calculating motions/positions, images captured via imagecapture device 848 may at step 1406 be digitally stabilized to counterthe detected motion by applying an EIS process. This EIS processing ofthe digital images may be performed in real time or near real time asthe images are captured and/or in post processing.

Also as previously mentioned, in some embodiments, the UAV 800 mayautonomously maneuver to stabilize capture by an image capture device848. Accordingly, in response to calculating motions/positions, systemsassociated with an localization and automated navigation system(described in more detail later) may at step 1407 generate commandsconfigured to cause the UAV 800 to execute flight maneuvers to countercertain detected motion.

Returning to the motorized gimbal, at step 1408 the control commandsand/or control signals are output to the motor(s) (e.g., motor(s) 842,844) to at step 1410 cause the motors to actuate one or more rotationjoints and thereby stabilize a mounted device (e.g., image capturedevice 848) relative to a particular frame of reference (e.g., thesurface of the Earth). As previously mentioned, in some embodiments themotor(s) may include integrated motor controller(s) (e.g., gimbal motorcontrollers 1907) and therefore may be configured to receive digitalcontrol commands generated by a separate processing unit (e.g.,processor 1912). In some embodiments, control signals in the form ofapplied voltage may be an output to induce electrical current within thestator coils of the motor(s).

Optionally, at step 1412, raw and/or processed sensor data may be runthrough a nonlinear estimator process (e.g., an extended Kalman filter)to produce more accurate position/motion estimations and reduce jitteror shakiness in the resulting active stabilization processes (e.g.,using motors, EIS, etc.). For example, calculated relativeposition/motion (e.g., by an IMU) can be based on a process commonlyreferred to as “dead reckoning.” In other words, a current position canbe continuously estimated based on previously estimated positions,measured velocity, and elapsed time. While effective to an extent, theaccuracy achieved through dead reckoning based on measurements from anIMU can quickly degrade due to the cumulative effect of errors in eachpredicted current position. Errors are further compounded by the factthat each predicted position is based on an calculated integral of themeasured velocity. To counter such effects, a nonlinear estimationalgorithm (one embodiment being an “extended Kalman filter”) may beapplied to a series of measured positions and/or orientations to producea real-time optimized prediction of the current position/motion based onassumed uncertainties in the observed data. Non-liner estimationprocessed such as Kalman filters are commonly applied in a number ofcontrol systems with feedback loops.

Also optionally, at step 1414, in some embodiments, the position motionof the motors(s) (i.e., angular position motion of the rotor axle(s))may be measured by one or more rotary encoders and this information maybe fed back into the process 1406 of generating controlcommands/signals. In some embodiments, as with the sensor data from themotion sensor(s), a nonlinear estimation process (e.g., Kalman filter)may be applied at step 1412 to the positional information output by therotary encoders before being used to generate the controlcommands/signals.

Note that the previously mentioned active systems have been described inthe context of stabilizing image capture to counter detected motion. Aperson having ordinary skill will recognize that similar systems (e.g.,motorize gimbal and/or digital image processing) can be applied torespond (directly or indirectly) to user control inputs. For example,gimbal motor controllers 1907 associated with a motorized gimbalmechanism 1954 may be configured to receive control commands based oninputs provided by a user such as a remote pilot of UAV 800 or anonboard pilot in a manned vehicle. Similarly, these systems can beapplied as part of an automated subject tracking system. For example,motor controllers associated with a motorized gimbal mechanism may beconfigured to receive control commands from a localization andnavigation system associated with UAV 800 to automatically track aparticular point in space or a detected physical object in thesurrounding environment.

Localization and Automated Navigation

FIG. 15 is a high-level illustration of a localization and navigationsystem 1500, according to some embodiments, for guiding navigation andimage/video capture by a UAV, for example UAV 800. The systems andmethods for automated localization and navigation are described hereinin the context of example UAV 800 for clarity and illustrative purposes.However, it shall be noted that UAV 800 may include fewer or moreautonomous navigation capabilities than as described. For example, insome embodiments UAV 800 may not include any of the autonomousnavigation capabilities described herein. According to some embodiments,a relative position and/or orientation of the UAV 800, one or moresubjects 1556, and/or one or more other physical objects in theenvironment surrounding UAV 800 may be determined using one or more ofthe subsystems illustrated in FIG. 15 . Further, this relative positionand/or orientation data may be used by the UAV 800 to autonomouslynavigate and to track subjects for image capture. The present teachinglocalization system 1500 may include an UAV 800, a GPS system comprisingmultiple GPS satellites 1502, a cellular system comprising multiplecellular antennae 1504 (with access to sources of localization data1506), a Wi-Fi system comprising multiple Wi-Fi routers 1508 (withaccess to sources of localization data 1506), and a portablemultifunction device (PMD) 1554 operated by a user 1552. Note, in FIG.15 the user 1552 is also the subject 1556 for image capture, however thesubject 1556 can also be any other real or virtual object or can be andefined point in space.

In some embodiments, PMD 1554 may include mobile, hand held or otherwiseportable computing devices that may be any of, but not limited to, anotebook, a laptop computer, a handheld computer, a palmtop computer, amobile phone, a cell phone, a PDA, a smart phone (e.g., iPhone®, etc.),a tablet (e.g., iPad®, etc.), a phablet (e.g., HTC Droid DNA™, etc.), atablet PC, a thin-client, a hand held console, a hand-held gaming deviceor console (e.g., XBOX®, etc.), mobile-enabled powered watch (e.g., iOS,Android or other platform based), a smart glass device (e.g., GoogleGlass™, etc.) and/or any other portable, mobile, hand held devices, etc.running on any platform or any operating system (e.g., OS X, iOS,Windows Mobile, Android, Blackberry OS, Embedded Linux platforms, PalmOS, Symbian platform, Google Chrome OS, etc.). A PMD 1554 may also be asimple electronic device comprising minimal components. For example, aPMD may simply include sensors for detecting motion and/or orientationand a transmitter/receiver means for transmitting and/or receiving data.

As mentioned earlier, a relative position and/or orientation of the UAV800, a relative position and/or orientation of the subject 1556, and/ora relative position and/or orientation of a PMD 1554 operated by a user1552 may be determined using one or more of the subsystems illustratedin FIG. 15 . For example, using only the GPS system 1502, a position onthe globe may be determined for any device comprising a GPS receiver(e.g., the UAV 800 and/or the PMD 1554). While GPS by itself in certainimplementations may provide highly accurate global positioning it isgenerally not capable of providing accurate information regardingorientation. Instead a technique of multiple inputs and multiple outputs(“MIMO”) (as illustrated in FIG. 15 ) may be used for localization,potentially in conjunction with other localization subsystems.

Consider the example based on the illustration in FIG. 15 ; a user 1552is utilizing an autonomous UAV 800 via a PMD 1554 to film herselfoverhead. In order to navigate the UAV 800 and inform the tracking by animage capture device (e.g., image capture device 848) of the subject1556 (in this case the user 1552), a relative position and orientationof the UAV 800 relative to the PMD 1554 (or any other point ofreference) may be necessary. This relative position between the UAV 800and the PMD 1554 may be determined using a GPS system to compare aglobal position of the UAV 800 and a global position of the PMD 1554.

Similarly, using an array of cellular and or/Wi-Fi antennae, a positionrelative to the known locations of antennae may be determined for boththe UAV 800 and PMD 1554 using known positioning techniques. Some knownpositioning techniques include those based on signal trilateration, forexample round trip time of arrival (RTT) in which a signal is sent andreceived by a signal transceiver and distance is calculated based on theelapsed time, received signal strength (RSS) in which the power levelsof the transmitted signal and the received signals are analyzed and adistance determined based on a known propagation loss. Other knownpositioning techniques include those based on signal triangulation, forexample angle of arrival (AoA) in which angles of arriving signals aredetermined and through applied geometry a position is determined.Current Wi-Fi standards, such as 803.11n and 802.11ac, allow for radiofrequency (RF) signal beamforming (i.e., directional signal transmissionusing phased-shifted antenna arrays) from transmitting Wi-Fi routers.Beamforming may be accomplished through the transmission of RF signalsat different phases from spatially distributed antennas (a “phasedantenna array”) such that constructive interference may occur at certainangles while destructive interference may occur at others, therebyresulting in a targeted directional RF signal field. Such a targetedfield is illustrated conceptually in FIG. 15 by dotted lines 1512emanating from Wi-Fi routers 1510.

As illustrated in FIG. 16 , a UAV 800 and/or PMD 1554 may include aphased array of Wi-Fi antenna and a relative position and/or pose may becalculated without the necessity for external existing Wi-Fi routers.According to some embodiments, the UAV 800 and/or PMD 1554 may transmitand/or receive a beamformed RF signal via a phased antenna array. TheUAV 800 and/or PMD 1554 may then detect the phase differences and powerlevels of the respective incoming signals and calculate an AoA for theincoming signals. For example, according to FIG. 16 , the PMD 1554 maydetermine an AoA of θ₁ for the RF signals 1602 transmitted by the UAV800. Similarly, the UAV 800 may determine an AoA of θ₂ for the RFsignals 1604 transmitted by the PMD 1554. This AoA information may thenbe incorporated with information gathered by an IMU on the UAV 100and/or PMD 104 (as well as other positioning data as described earlier)in order to infer a relative position and/pose between the UAV 800 andthe PMD 1554.

According to some embodiments, an array of Wi-Fi transmitters and signalmonitors may be utilized for device-free passive localization of objectsthat are not transmitting signals (e.g., a human subject 1556 notcarrying a PMD 1554). FIG. 17 illustrates an example system 1700 fordevice-free passive localization of a subject (e.g., a human subject1556). In this example a human subject 1556 passes through a network ofWi-Fi transmitters 1708 transmitting RF signals. The signal monitors1710 (e.g., standard wireless sniffers) may detect changes in thecharacteristics of the RF signals received from the Wi-Fi transmitters1708 caused by interference as the human subject 1556 passes through thesignal field. Using localization algorithms, such changes in the RFsignal field may be correlated to the presence of an object, its type,its orientation and its location. Also, according to FIG. 17 ,information gathered by device-free passive localization system 1700 maybe fed wirelessly (e.g., via Wi-Fi connection 1730) to a nearby UAV 800in order to inform its tracking of the human subject 1556.

According to some embodiments, an inertial measurement unit (IMU) may beused to determine relative position and/or orientation. An IMU is adevice that calculates a vehicle's velocity, orientation, andgravitational forces using a combination of accelerometers andgyroscopes. As described herein, an UAV 800 and/or PMD 1554 may includeone or more IMUs. Using a method commonly referred to as “deadreckoning” an IMU (or associated systems) may be used to calculate andtrack a predicted position based on a previously known position(s) usingmeasured velocities and the time elapsed from the previously knownposition(s). While effective to an extent, the accuracy achieved throughdead reckoning based on measurements from an IMU quickly degrades due tothe cumulative effect of errors in each predicted current position.Errors are further compounded by the fact that each predicted positionis based on an calculated integral of the measured velocity. To countersuch effects, an embodiment utilizing localization using an IMU mayinclude localization data from other sources (e.g., the GPS, Wi-Fi, andcellular systems described above) to continuously update the last knownposition and/or orientation of the object. Further, a nonlinearestimation algorithm (one embodiment being an “extended Kalman filter”)may be applied to a series of measured positions and/or orientations toproduce a real-time optimized prediction of the current position and/ororientation based on assumed uncertainties in the observed data. Kalmanfilters are commonly applied in the area of aircraft navigation,guidance, and controls.

According to some embodiments, computer vision may be used to determinea relative position and/or orientation of a UAV 800 or any other object.The term, “computer vision” in this context may generally refer to theacquiring, processing, analyzing and understanding of captured images.Consider again the localization system 1500 illustrated in FIG. 15 .According to some embodiments, UAV 800 may include image capture devicesand computer vision capabilities. In this example, UAV 100 may beprogramed to track a subject (e.g., a human or some other object). Usingcomputer vision, UAV 800 may recognize the subject in images captured bythe image capture devices and may use the recognition information toperform aerial maneuvers to keep the subject in view, and/or may makeadjustments in image capture (e.g., using a gimbaled image capturedevice) to keep the subject in view.

Relative position and/or orientation may be determined through computervision using a number of methods. According to some embodiments an imagecapture device of the UAV 800 may include two or more cameras. Bycomparing the captured image from two or more vantage points, a systememploying computer vision may calculate a distance to a capturedphysical object. With the calculated distance as well as other positionand/or orientation data for the UAV (e.g., data from GPS, Wi-Fi,Cellular, and/or IMU, as discussed above) a relative position and/ororientation may be determined between the UAV 800 and a point ofreference (e.g., the captured physical object).

According to some embodiments, an image capture device of UAV 800 may bea single camera (i.e., a non-stereoscopic camera). Here, computer visionalgorithms may identify the presence of an object and identify theobject as belonging to a known type with particular dimensions. Forexample, through computer vision, the object may be identified as anadult male human. With this recognition data, as well as other positionand/or orientation data for the UAV 100 (e.g., data from GPS, Wi-Fi,Cellular, and/or IMU, as discussed above), UAV 100 may predict arelative position and/or orientation of the object.

According to some embodiments, computer vision may be used along withmeasurements from an IMU (or accelerometer(s) or gyroscope(s)) withinthe UAV and/or PMD 1554 carried by a user 1552 as illustrated in FIG.18A-18B. FIG. 18A shows a simple diagram that illustrates how sensordata gathered by an IMU at a PMD 15544 may be applied to sensor datagathered by an image capture device at an UAV 800 to determine positionand/or orientation data of a physical object (e.g., a user 1552).Outline 1850 represents a 2-dimensional image captured field of view atUAV 800. As shown in FIG. 18A, the field of view 1850 includes the imageof a physical object (e.g., user 1552) moving from one position toanother. From its vantage point, UAV 800 may determine a distance Atraveled across the image capture field of view 1850. The PMD 1554,carried by user 1552, may determine an actual distance B traveled by theuser 1552 based on measurements by internal sensors (e.g., the IMU) andan elapsed time. The UAV 800 may then receive the sensor data and/or thedistance B calculation from PMD 1554 (e.g., via wireless RF signal).Correlating the difference between the observed distance A and thereceived distance B, UAV 800 may determine a distance D between UAV 800and the physical object (e.g., user 1552). With the calculated distanceas well as other position and/or orientation data for the UAV 800 (e.g.,data from GPS, Wi-Fi, Cellular, and/or IMU, as discussed above) arelative position and/or orientation may be determined between the UAV800 and the physical object (e.g., user 1552).

Alternatively, estimations for the position and/or orientation of eitherthe UAV 800 or PMD 1554 may be made using a process generally referredto as “visual inertial odometry” or “visual odometry.” FIG. 18Billustrates the working concept behind visual odometry at a high level.A plurality of images is captured in sequence as a camera moves throughspace. Due to the movement of the camera, the images captured of thesurrounding space change from frame to frame. In FIG. 18B, this isillustrated by initial image capture field of view 1852 and a subsequentimage capture field of view 1854 captured as the image capture devicehas moved from a first position and orientation to a second position andorientation over an elapsed time. In both images, the image capturedevice may capture real world physical objects, for example, the house1880 and/or a human subject 1556 (e.g., user 1552). Computer visiontechniques are applied to the sequence of images to detect and matchfeatures of physical objects captured in the field of view of thecamera. For example, in FIG. 18B, features such as the head of a humansubject or the corner of the chimney on the house 1880 are identified,matched, and thereby tracked. By incorporating sensor data from an IMU(or accelerometer(s) or gyroscope(s)) associated with the camera to thetracked features of the image capture, estimations may be made for theposition and/or orientation of the camera over time. This technique maybe applied at both the UAV 800 and PMD 15544 to calculate the positionand/or orientation of both systems. Further, by communicating theestimates between the systems (e.g., via a Wi-Fi connection) estimatesmay be calculated for the respective positions and/or orientationsrelative to each other. As previously mentioned position, orientation,and motion estimation based in part on sensor data from an on board IMUmay introduce error propagation issues. As previously stated,optimization techniques may be applied to position, orientation, andmotion estimations to counter such uncertainties. In some embodiments, anonlinear estimation algorithm (one embodiment being an “extended Kalmanfilter”) may be applied to a series of measured positions and/ororientations to produce a real-time optimized prediction of the currentposition and/or orientation based on assumed uncertainties in theobserved data.

According to some embodiments, computer vision may include remotesensing technologies such as laser illuminated detection and ranging(LIDAR or Lidar). For example, an UAV 800 equipped with LIDAR may emitone or more laser beams in a continuous scan up to 360 degrees in alldirections around the UAV 800. Light received by the UAV 800 as thelaser beams reflect off physical objects in the surrounding physicalworld may be analyzed to construct a real time 3D computer model of thesurrounding physical world. Such 3D models may be analyzed to identifyparticular physical objects (e.g., a user 1552) in the physical worldfor tracking. Further, images captured by an image capture device may becombined with the laser constructed 3D models to form textured 3D modelsthat may be further analyzed in real time or near real time for physicalobject recognition (e.g., by using computer vision algorithms).

The computer vision-aided localization and navigation system describedabove may calculate the position and/or orientation of features in thephysical world in addition to the position and/or orientation of the UAV800 and/or PMD 1554. The position of these features may then be fed intothe navigation system such that motion trajectories may be planned thatavoid obstacles. In addition, in some embodiments, the visual navigationalgorithms may incorporate data from proximity sensors (e.g.,electromagnetic, acoustic, and/or optics based) to estimate obstacleposition with more accuracy. Further refinement may be possible with theuse of stereoscopic computer vision with multiple cameras, as describedearlier.

According to some embodiments, the previously described relativeposition and/or orientation calculations may be performed by an UAV 800,PMD 1554, remote computing device(s) (not shown in the figures), or anycombination thereof.

The localization system 1500 of FIG. 15 (including all of the associatedsubsystems as previously described) is only one example of a system forlocalization and navigation. Localization system 1500 may have more orfewer components than shown, may combine two or more components, or amay have a different configuration or arrangement of the components.Some of the various components shown in FIGS. 15 through 18B may beimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits.

Unmanned Aerial Vehicle—System Components

An Unmanned Aerial Vehicle (UAV), sometimes referred to as a drone, isgenerally defined as any aircraft capable of controlled flight without ahuman pilot onboard. UAVs may be controlled autonomously by onboardcomputer processors and/or via remote control by a remotely locatedhuman pilot. Similar to an airplane, UAVs may utilize fixed aerodynamicsurfaces along means for propulsion (e.g., propeller, rotor, jet. etc.)to achieve lift. Alternatively, similar to helicopters, a UAV maydirectly use means for propulsion (e.g., propeller, rotor, jet. etc.) tocounter gravitational forces and achieve lift. Propulsion-driven lift(as in the case of helicopters) offers significant advantages in certainimplementations, for example as a mobile filming platform, because itallows for controlled motion along all axes.

Multi-rotor helicopters, in particular quadcopters, have emerged as apopular UAV configuration. A quadcopter (also known as a quadrotorhelicopter or quadrotor) is a multi-rotor helicopter that is lifted andpropelled by four rotors. Unlike most helicopters, quadcopters use twosets of two fixed-pitch propellers. A first set of rotors turnsclockwise, while a second set of rotors turns counter-clockwise. Inturning opposite directions, the first set of rotors may counter theangular torque caused by the rotation of the other set, therebystabilizing flight. Flight control is achieved through variation in theangular velocity of each of the four fixed-pitch rotors. By varying theangular velocity of each of the rotors, a quadcopter may perform preciseadjustments in its position (e.g., adjustments in altitude and levelflight left, right, forward and backward) and orientation, includingpitch (rotation about a first lateral axis), roll (rotation about asecond lateral axis), and yaw (rotation about a vertical axis). Forexample, if all four rotors are spinning (two clockwise, and twocounter-clockwise) at the same angular velocity, the net aerodynamictorque about the vertical yaw axis is zero. Provided the four rotorsspin at sufficient angular velocity to provide a vertical thrust equalto the force of gravity, the quadcopter can maintain a hover. Anadjustment in yaw may be induced by varying the angular velocity of asubset of the four rotors thereby mismatching the cumulative aerodynamictorque of the four rotors. Similarly, an adjustment in pitch and/or rollmay be induced by varying the angular velocity of a subset of the fourrotors but in a balanced fashion such that lift is increased on one sideof the craft and decreased on the other side of the craft. An adjustmentin altitude from hover may be induced by applying a balanced variationin all four rotors thereby increasing or decreasing the vertical thrust.Positional adjustments left, right, forward, and backward may be inducedthrough combined pitch/roll maneuvers with balanced applied verticalthrust. For example, to move forward on a horizontal plane, thequadcopter would vary the angular velocity of a subset of its fourrotors in order to perform a pitch forward maneuver. While pitchingforward, the total vertical thrust may be increased by increasing theangular velocity of all the rotors. Due to the forward pitchedorientation, the acceleration caused by the vertical thrust maneuverwill have a horizontal component and will therefore accelerate the craftforward on horizontal plane.

FIG. 19 is a high-level diagram illustrating a system 1900 of componentsof example UAV 800, according to some embodiments. UAV system 1900 mayinclude several subsystems. For example, UAV system 1900 may include oneor more propulsion systems 1952. As shown in FIG. 19 , in an embodiment,propulsion system 1952 includes one or more means for propulsion (e.g.,rotors 1902 and motor(s) 1904) and one or more electronic speedcontrollers 1906 configured to regulate power to the means ofpropulsion. UAV system 1900 may also include a motorized gimbal system1954 that includes gimbal motor controllers 1907 and gimbal motors 1901(e.g., similar to previously described motors 842 and 844). UAV system1900 may also include a flight controller 1908, a peripheral interface1910, a processor(s) 1912, a memory controller 1914, a memory 1916(which may include one or more computer readable storage mediums), apower module 1918, a GPS module 1920, a communications interface 1922,an audio circuitry 1924, an accelerometer 1926 (including subcomponentssuch as gyroscopes), an inertial measurement unit (IMU) 1928, aproximity sensor 1930, an optical sensor controller 1932 and associatedoptical sensor(s) 1934, a portable multifunction device (PMD) interfacecontroller 1936 with associated interface device(s) 1938, and any otherinput controllers 1940 and input device 1942, for example displaycontrollers with associated display device(s). General terms such as“sensors” may refer to one or more components or combinations ofcomponents, for example, microphone 1924, proximity sensors 1930,accelerometers 1926, an inertial measurement unit (IMU) 1928, opticalsensors 1934, and any combination thereof. These components maycommunicate over one or more communication buses, interconnects, wires,or signal lines as represented by the arrows in FIG. 19 .

UAV system 1900 is only one example of a system for use in UAV 800. UAVsystem 1900 may have more or fewer components than shown, may combinetwo or more components as functional units, or a may have a differentconfiguration or arrangement of the components. Some of the variouscomponents shown in FIG. 19 may be implemented in hardware, software, ora combination of both hardware and software, including one or moresignal processing and/or application specific integrated circuits.

As described earlier, the propulsion system 1952 may include afixed-pitch rotor. The propulsion system 1952 may also include avariable-pitch rotor (for example, using a gimbal mechanism), avariable-pitch jet engine, or any other mode of propulsion having theeffect of providing force. The means for propulsion system 1952 mayinclude a means for varying the applied thrust, for example via anelectronic speed controller 1906 varying the speed of each fixed-pitchrotor.

Flight Controller 1908 (sometimes referred to as a “flight controlsystem” or “autopilot”) may include a combination of hardware and/orsoftware configured to receive input data (e.g., input control commandsfrom PMD 1554 and or sensor data from an accelerometer 1926 or 1928),interpret the data and output control signals to the propulsion system1952 and/or aerodynamic surfaces (e.g., fixed wing control surfaces) ofthe UAV 800. Alternatively, or in addition, a flight controller 1908 maybe configured to receive control commands generated by another componentor device (e.g., processors 1912 and/or a separate remote computingdevice), interpret those control commands and generate control signalsto propulsion system 1952. In some embodiments, a flight controller 1908may be integrated with propulsion system 1952 as a single modular unitconfigured to receive control commands from a separate processing unit1912.

Motorized gimbal mechanism 1954 may be part of an image capture assembly840, as described previously. The gimbal motor controller(s) 1907 ofsystem 1954 may include a combination of hardware and/or softwareconfigured to receive input sensor data (e.g., from an accelerometer1926 or IMU 1928), interpret the data and output control signals to themotor(s) 604 of the motorized gimbal 100. Alternatively, or in addition,a gimbal motor controller 1907 may be configured to receive controlcommands generated by another component or device (e.g., processors 1912and/or a separate remote computing device), interpret those controlcommands and generate control signals to the gimbal motor(s) 1901 of themotorized gimbal mechanism 1954. In some embodiments, a gimbal motorcontroller 1907 may be integrated with a gimbal motor 1901 as a singlemodular unit configured to receive control commands from a separateprocessing unit 1912.

Memory 1916 may include high-speed random-access memory and may alsoinclude non-volatile memory, such as one or more magnetic disk storagedevices, flash memory devices, or other non-volatile solid-state memorydevices. Access to memory 1916 by other components of UAV system 1900,such as the processors 1912 and the peripherals interface 1910, may becontrolled by the memory controller 1914.

The peripherals interface 1910 may couple the input and outputperipherals of the UAV 800 to the processor(s) 1912 and memory 1916. Theone or more processors 1912 run or execute various software programsand/or sets of instructions stored in memory 1916 to perform variousfunctions for the UAV 800 and to process data. In some embodiments,processors 1912 may include general central processing units (CPUs),specialized processing units such as Graphical Processing Units (GPUs)particularly suited to parallel processing applications, or anycombination thereof.

In some embodiments, the peripherals interface 1910, the processor(s)1912, and the memory controller 1914 may be implemented on a singleintegrated chip. In some other embodiments, they may be implemented onseparate chips.

The network communications interface 1922 may facilitate transmissionand reception of communications signals often in the form ofelectromagnetic signals. The transmission and reception ofelectromagnetic communications signals may be carried out over physicalmedia such copper wire cabling or fiber optic cabling, or may be carriedout wirelessly for example, via a radiofrequency (RF) transceiver. Insome embodiments the network communications interface may include RFcircuitry. In such embodiments, RF circuitry may convert electricalsignals to/from electromagnetic signals and communicate withcommunications networks and other communications devices via theelectromagnetic signals. The RF circuitry may include well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. The RFcircuitry may facilitate transmission and receipt of data overcommunications networks (including public, private, local, and widearea). For example, communication may be over a wide area network (WAN),a local area network (LAN), or a network of networks such as theInternet. Communication may be facilitated over wired transmission media(e.g., via Ethernet) or wirelessly. Wireless communication may be over awireless cellular telephone network, a wireless local area network (LAN)and/or a metropolitan area network (MAN), and other modes of wirelesscommunication. The wireless communication may use any of a plurality ofcommunications standards, protocols and technologies, including but notlimited to Global System for Mobile Communications (GSM), Enhanced DataGSM Environment (EDGE), high-speed downlink packet access (HSDPA),wideband code division multiple access (W-CDMA), code division multipleaccess (CDMA), time division multiple access (TDMA), Bluetooth, WirelessFidelity (Wi-Fi) (e.g., IEEE 802.11ac), voice over Internet Protocol(VoIP), Wi-MAX, or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

The audio circuitry 1924, including the speaker and microphone 1950 mayprovide an audio interface between the surrounding environment and theUAV 800. The audio circuitry 1924 may receive audio data from theperipherals interface 1910, convert the audio data to an electricalsignal, and transmits the electrical signal to the speaker 1950. Thespeaker 1950 may convert the electrical signal to human-audible soundwaves. The audio circuitry 1924 may also receive electrical signalsconverted by the microphone 1950 from sound waves. The audio circuitry1924 may convert the electrical signal to audio data and transmits theaudio data to the peripherals interface 1910 for processing. Audio datamay be retrieved from and/or transmitted to memory 1916 and/or thenetwork communications interface 1922 by the peripherals interface 1910.

The I/O subsystem 1960 may couple input/output peripherals on the UAV800, such as an optical sensor system 1934, the PMD interface device1938, and other input/control devices 1942, to the peripherals interface1910. The I/O subsystem 1960 may include an optical sensor controller1932, a PMD interface controller 1936, and other input controller(s)1940 for other input or control devices. The one or more inputcontrollers 1940 receive/send electrical signals from/to other input orcontrol devices 1942.

The other input/control devices 1942 may include physical buttons (e.g.,push buttons, rocker buttons, etc.), dials, touch screen displays,slider switches, joysticks, click wheels, and so forth. A touch screendisplay may be used to implement virtual or soft buttons and one or moresoft keyboards. A touch-sensitive touch screen display may provide aninput interface and an output interface between the UAV system 1900 anda user. A display controller may receive and/or send electrical signalsfrom/to the touch screen. The touch screen may display visual output tothe user. The visual output may include graphics, text, icons, video,and any combination thereof (collectively termed “graphics”). In someembodiments, some or all of the visual output may correspond touser-interface objects, further details of which are described below.

A touch sensitive display system may have a touch-sensitive surface,sensor or set of sensors that accepts input from the user based onhaptic and/or tactile contact. The touch sensitive display system andthe display controller (along with any associated modules and/or sets ofinstructions in memory 1916) may detect contact (and any movement orbreaking of the contact) on the touch screen and convert the detectedcontact into interaction with user-interface objects (e.g., one or moresoft keys or images) that are displayed on the touch screen. In anexemplary embodiment, a point of contact between a touch screen and theuser corresponds to a finger of the user.

The touch screen may use LCD (liquid crystal display) technology, or LPD(light emitting polymer display) technology, although other displaytechnologies may be used in other embodiments. The touch screen and thedisplay controller may detect contact and any movement or breakingthereof using any of a plurality of touch sensing technologies now knownor later developed, including but not limited to capacitive, resistive,infrared, and surface acoustic wave technologies, as well as otherproximity sensor arrays or other elements for determining one or morepoints of contact with a touch screen.

The PMD interface device 1938 along with PMD interface controller 1936may facilitate the transmission of data between the UAV system 1900 anda PMD 1554. According to some embodiments, communications interface 1922may facilitate the transmission of data between UAV 800 and a PMD 1554(for example where data is transferred over a local Wi-Fi network).

The UAV system 1900 also includes a power system 1918 for powering thevarious components. The power system 1918 may include a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in computerized device.

The UAV system 1900 may also include one or more optical sensors 1934.FIG. 19 shows an optical sensor coupled to an optical sensor controller1932 in I/O subsystem 1960. The optical sensor 1934 may include acharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. The optical sensor 1934 receives light from theenvironment, projected through one or more lens (the combination ofoptical sensor and lens herein referred to as a “camera”) and convertsthe light to data representing an image. In conjunction with an imagingmodule located in memory 1916, the optical sensor 1932 may capture stillimages and/or video. Optical sensors 1934 may be understood as the sameor similar as image capture devices 884 described with respect to FIGS.8A and 8D and gimbaled image capture device 848 described with respectto FIGS. 11A-11F.

The UAV system 1900 may also include one or more proximity sensors 1330.FIG. 13 shows a proximity sensor 1330 coupled to the peripheralsinterface 1310. Alternately, the proximity sensor 1330 may be coupled toan input controller 1340 in the I/O subsystem 1360. Proximity sensors1330 may generally include remote sensing technology for proximitydetection, range measurement, target identification, etc. For example,proximity sensors 1330 may include radar, sonar, and light illuminateddetection and ranging (Lidar).

The UAV system 1900 may also include one or more accelerometers 1926.FIG. 19 shows an accelerometer 1926 coupled to the peripherals interface1910. Alternately, the accelerometer 1926 may be coupled to an inputcontroller 1940 in the I/O subsystem 1960.

The UAV system 1900 may include one or more inertial measurement units(IMU) 1928. An IMU 1928 may measure and report the UAV's velocity,acceleration, orientation, and gravitational forces using a combinationof gyroscopes and accelerometers (e.g., accelerometer 1926). Aspreviously mentioned, accelerometers 1926 and IMU 1928 may be mounted todifferent components of UAV 800. For example, accelerometers 1926 and/orIMU 1928 can be mounted to any of housing 810, passive stabilizationassembly 830, motors 842, 844, or image capture device 848 to detectmotion in different frames of reference.

The UAV system 1900 may include a global positioning system (GPS)receiver 1920. FIG. 19 shows an GPS receiver 1920 coupled to theperipherals interface 1310. Alternately, the GPS receiver 1920 may becoupled to an input controller 1940 in the I/O subsystem 1960. The GPSreceiver 1320 may receive signals from GPS satellites in orbit aroundthe earth, calculate a distance to each of the GPS satellites (throughthe use of GPS software), and thereby pinpoint a current global positionof UAV 800. In some embodiments, positioning of UAV 800 may beaccomplished without GPS satellites through the use of other techniquesas described herein.

In some embodiments, the software components stored in memory 1916 mayinclude an operating system, a communication module (or set ofinstructions), a flight control module (or set of instructions), alocalization module (or set of instructions), a computer vision module,a graphics module (or set of instructions), and other applications (orsets of instructions). For clarity one or more modules and/orapplications may not be shown in FIG. 19 .

The operating system (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VxWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

A communications module may facilitate communication with other devicesover one or more external ports 1944 and may also include varioussoftware components for handling data transmission via the networkcommunications interface 1922. The external port 1944 (e.g., UniversalSerial Bus (USB), FIREWIRE, etc.) may be adapted for coupling directlyto other devices or indirectly over a network (e.g., the Internet,wireless LAN, etc.).

A graphics module may include various software components forprocessing, rendering and displaying graphics data. As used herein, theterm “graphics” may include any object that can be displayed to a user,including without limitation text, still images, videos, animations,icons (such as user-interface objects including soft keys), and thelike. The graphics module in conjunction with a graphics processing unit(GPU) 1912 may process in real time or near real time, graphics datacaptured by optical sensor(s) 1934 and/or proximity sensors 1930.

A computer vision module, which may be a component of graphics module,provides analysis and recognition of graphics data. For example, whileUAV 800 is in flight, the computer vision module along with graphicsmodule (if separate), GPU 1912, and optical sensor(s) 1934 and/orproximity sensors 1930 may recognize and track the captured image of asubject located on the ground. The computer vision module may furthercommunicate with a localization/navigation module and flight controlmodule to update a relative position between UAV 800 and a point ofreference, for example a target object (e.g., a PMD or human subject),and provide course corrections to maintain a constant relative positionwhere the subject is in motion.

A localization/navigation module may determine the location and/ororientation of UAV 800 and provides this information for use in variousmodules and applications (e.g., to a flight control module in order togenerate commands for use by the flight controller 1908).

An active image capture stabilization module may process motioninformation (e.g., from sensors 1926, 1928) to generate (e.g., using ausing a multi-axis stabilization algorithm) control signals/commandsconfigured to control gimbal motor(s) 1901. Similarly, active imagecapture stabilization module may process motion information (e.g., fromsensors 1926, 1928) to digitally stabilized captured images (e.g., viaan optical sensor device 1934) using an EIS process. An examplestabilization process that optionally incorporates a feedback loop isdescribed at a high level with respect to FIG. 14 .

Optical sensor(s) 1934 in conjunction with, optical sensor controller1932, and a graphics module, may be used to capture still images orvideo (including a video stream) and store them into memory 1916.

Each of the above identified modules and applications correspond to aset of instructions for performing one or more functions describedabove. These modules (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 1916 maystore a subset of the modules and data structures identified above.Furthermore, memory 1916 may store additional modules and datastructures not described above.

What is claimed is:
 1. An aerial vehicle comprising: a housing; an imagecapture assembly arranged proximate to a front end of the housing; andan image stabilization assembly configured to isolate the image captureassembly from vibration of the housing using an elongated arm weightedto counterbalance a mass of components of the image capture assembly,wherein the elongated arm extends into an interior space of the housingfrom the image capture assembly toward a back end of the housing.
 2. Theaerial vehicle of claim 1, further comprising: one or more passivevibration isolators, wherein the elongated arm comprises a distal endand a proximal end, the distal end coupled to an interior surface of thehousing via one or more passive vibration isolators and the proximal endcoupled to the image capture assembly.
 3. The aerial vehicle of claim 2,wherein at least one of the one or more passive vibration isolatorscomprises an elastic vibration isolator.
 4. The aerial vehicle of claim2, wherein at least one of the one or more passive vibration isolatorshas a 1:1:1 ratio of compression stiffness to tensile stiffness to shearstiffness.
 5. The aerial vehicle of claim 2, further comprising: amounting assembly, wherein the proximal end of the elongated arm iscoupled to the image capture assembly via the mounting assembly.
 6. Theaerial vehicle of claim 1, wherein the image stabilization assembly isconfigured to isolate the image capture assembly from vibration of thehousing when the vibration is above a threshold frequency.
 7. The aerialvehicle of claim 6, wherein the threshold frequency is 15 Hz.
 8. Theaerial vehicle of claim 1, wherein the image stabilization assemblycomprises: an active stabilization assembly including a motorizedgimbal, the active stabilization assembly configured to actively countermotion by the aerial vehicle to actively stabilize the imagestabilization assembly; and a passive stabilization assembly includingthe elongated arm, the passive stabilization configured to passivelycounter the motion by the aerial vehicle to passively stabilize theimage stabilization assembly.
 9. The aerial vehicle of claim 1, whereinthe elongated arm is made of one or more of plastic, metal, carbonfiber, synthetic fiber, or a composite material.
 10. An imagestabilization apparatus for isolating an image capture assembly fromvibration of a housing, the apparatus comprising: one or more passivevibration isolators; and an elongated arm weighted to counterbalance aweight of the image capture assembly, the elongated arm having aproximal end adapted to couple with the image capture assembly and adistal end coupled to the housing via the one or more passive vibrationisolators, wherein the image capture assembly includes an electronicimaging stabilization (EIS) system configured to receive and processimages captured by the image capture device to actively counter adetected motion.
 11. The image stabilization apparatus of claim 10,wherein at least one of the one or more passive vibration isolatorscomprises an elastic vibration isolator.
 12. The image stabilizationapparatus of claim of 10, wherein at least one of the one or morepassive vibration isolators has a 1:1:1 ratio of compression stiffnessto tensile stiffness to shear stiffness.
 13. The image stabilizationapparatus of claim of 10, further comprising: a mounting assembly,wherein the proximal end of the elongated arm is coupled to the imagecapture assembly via the mounting assembly.
 14. The image stabilizationapparatus of claim of 10, wherein the housing is coupled to an unmannedaerial vehicle (UAV).
 15. The image stabilization apparatus of claim of14, wherein the housing has a front end and a back end, the imagecapture assembly is arranged proximate to the front end of the housing,and the elongated arm extends into an interior space of the housing fromthe image capture assembly towards the back end of the housing.
 16. Theimage stabilization apparatus of claim of 10, wherein the imagestabilization assembly is configured to isolate the image captureassembly from vibration of the housing when the vibration is above athreshold frequency.
 17. A system for isolating an image captureassembly from vibration of a housing, the system comprising: one or morepassive vibration isolators; an apparatus coupled to the housing via theone or more passive vibration isolators; a mounting assembly coupled tothe apparatus and adapted to couple the apparatus with the image captureassembly; wherein the apparatus is weighted to counterbalance a mass ofthe components of the image capture assembly, and wherein the imagecapture assembly includes an electronic imaging stabilization (EIS)system configured to receive and process images captured by the imagecapture device to actively counter a detected motion.
 18. The system ofclaim 17, wherein the housing is coupled to an unmanned aerial vehicle(UAV), wherein the image capture assembly includes an image capturedevice configured to capture images of a surrounding physicalenvironment while UAV is in flight.
 19. The system of claim 17, whereinat least one of the one or more passive vibration isolators has a 1:1:1ratio of compression stiffness to tensile stiffness to shear stiffness.20. The system of claim 17, wherein the system is configured to isolatethe image capture assembly from vibration of the housing that is above athreshold frequency.